[Medium] Per-chunk IPC collective count is now dtype-content-dependent → deadlock if ranks' dtype composition diverges.

iter_single_dtype_chunks yields one sub-chunk per distinct dtype (first-seen order), and _send_single_dtype_chunk_vllm_native issues an unconditional all_gather_object + 2 barrier per sub-chunk. So the per-logical-chunk collective count now equals the number of distinct dtypes in that chunk on that rank. Pre-PR this count was purely structural (bucket layout, identical across ranks); it is now content-dependent. If any rank's logical chunk has a different dtype set/order (e.g. weight+scale on one rank vs weight-only on another), ranks call all_gather_object a different number of times and hang.

This relies on extract_weights (out of scope) producing perfectly rank-symmetric dtype composition; the transport takes it on faith. The intra-sub-chunk if any(tensor.dtype != dtype): raise guard is dead code and does not protect against cross-rank divergence.

Fix: before the loop, all_gather_object each rank's per-logical-chunk sub-chunk dtype list and assert equality (fail fast instead of hanging), or drive the partition from a rank-0 broadcast. At minimum document the invariant.

[Medium] Power-of-two (Blackwell) serialized path neither defaults nor validates VLLM_USE_DEEP_GEMM_E8M0 — asymmetric with the FP32 path → silent numeric mismatch.

This scale_mode == "1" (FP32) branch both rejects a conflicting VLLM_USE_DEEP_GEMM_E8M0 != "0" and pins the default to "0", so TE and vLLM agree. The scale_mode == "0" power-of-two branch (the documented Blackwell case) does neither — the var is only forwarded if already present in the driver env.

Scenario: Blackwell run sets NVTE_FP8_BLOCK_SCALING_FP32_SCALES=0 + serialized_blockwise, fp8_param=false (passes all config checks) but forgets VLLM_USE_DEEP_GEMM_E8M0=1. TE emits E8M0 power-of-two scales; vLLM interprets them without E8M0 → mis-dequantized weights → silently degraded rollouts, no error. The symmetric FP32 mistake is caught.

Fix: handle scale_mode == "0" symmetrically — default VLLM_USE_DEEP_GEMM_E8M0=1 and reject a conflicting != "1".

@YJHMITWEB this seems correct, shouldn't export VLLM_USE_DEEP_GEMM_E8M0 be 1 for blackwell? or is there some reason that it doesn't work w/ blockwise + power of 2?

not seeing VLLM_USE_DEEP_GEMM_E8M0=1 in the config that you're using as well so curious about this

@YJHMITWEB this seems correct, shouldn't export VLLM_USE_DEEP_GEMM_E8M0 be 1 for blackwell? or is there some reason that it doesn't work w/ blockwise + power of 2?

not seeing VLLM_USE_DEEP_GEMM_E8M0=1 in the config that you're using as well so curious about this

this 128 number is specific to blockwise quantization right? can we clarify the wording to reflect that

can we update this config in https://github.com/NovaSky-AI/SkyRL/blob/main/docs/content/docs/configuration/config.mdx as well

quick question for my understanding: so this is a custom environment variable that we introduced? I can't find it anywhere inside TransformerEngine or elsewhere in OSS

And this is needed because with power of 2 scaling (which is needed on blackwell) + blockwise is not well supported in TE, and doesn't handle CP/TP packing where we can get a block of all 0s

And the reason existing recipes on TransformerEngine using Float8BlockScaling are able to run without issue is because they all use hopper? And everyone is primarily is using MXFP8BlockScaling on blackwell where there is no amax_epsilon concept?

so the problem here is that:

if fp8 param is enabled, then megatron bridge will not store the unquantized weights?

looking through megatron bridge i'm seeing that on to_megatron_provider if load_weights=true, then we call load_weights_hf_to_megatron, which has the following logic to store the unquantized state dict in the bridge: https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/47254ca622d35b8063fcc780b070588836ddf20a/src/megatron/bridge/models/conversion/model_bridge.py#L1072

is there a reason we need to defer it from here to manually calling self.bridge.load_hf_weights(self.actor_module)? might be missing something, pls correct if my understanding is wrong? Otherwise i think the _release_fp8_param_unquantized_state and initialize_fp8_param_optimizer_masters makes sense to me since we do need to pass that state dict to initialize the optimizer state.

qq: if we have fp8 param on, are the quantization operations here no-ops? or are we unconditionally re-quantizing?

if so should we handle fp8 param case separately? (potentially using the interface from this pr: NVIDIA-NeMo/Megatron-Bridge#1994 )

one high level thought: there is this PR in megatron-bridge: NVIDIA-NeMo/Megatron-Bridge#1994 that was merged more than 3 months ago that introduces a new function: build_export_fp8_tasks - this is designed for the case where we have fp8 params, and allows the user to ship fp8 params + scale metadata directly to vLLM

ideally we could push down the logic in this file for exporting an iterable list of fp8 params to Megatron-Bridge as well and just add support for the case where we have bf16 params but we want to export them to fp8.

mentioned this in the previous comment about re-organizing files, but these mappings should definitely be in a separate file if we have them at all, and the references to fp8 weight suffixes, up/down suffixes, and unquantized suffixes should be made generic behind some abstraction.

Ideally which layers are quantized could be configured, but we can leave that as a follow up of this PR. we should at least design the interface that defines the quantization spec rather than hard coding.

Overall some abstraction like this:

@dataclass(frozen=True)
class ModelFp8Spec:
    matches: Callable[[Any], bool]              # hf_config -> bool (replaces is_qwen35_config)
    should_quantize: Callable[[str, Sequence[int]], bool]   # name/shape -> quantize?
    ignored_layers: Callable[[Any], list[str]]  # hf_config -> vLLM ignored modules (residual shape/vLLM denylist, e.g. vision proj)
    moe_expert_spec: Callable[[str], MoeExpertSpec | None]  # gate/up split + expert batching

@dataclass(frozen=True)
class MoeProjection:
    hf_name: str          # "gate_proj" | "up_proj" | "down_proj"  (what the sender emits on the wire)
    vllm_param: str       # ".experts.w13_weight" | ".experts.w2_weight"  (vLLM fused target, weight only)
    shard_id: str         # "w1" | "w3" | "w2"
    split: Optional[tuple[int, int]] = None   # (chunk_index, num_chunks) along split_dim; None = whole tensor

@dataclass(frozen=True)
class MoeExpertSpec:
    source_suffix: str    # ".mlp.experts.gate_up_proj" | ".mlp.experts.down_proj"  (Bridge-exported batched name)
    split_dim: int        # dim to carve projections from the fused [E, ...] tensor (1 = per-expert output rows)
    projections: tuple[MoeProjection, ...]

QWEN35_MOE_EXPERT_SPECS = (
    MoeExpertSpec(
        source_suffix=".mlp.experts.gate_up_proj",
        split_dim=1,
        projections=(
            MoeProjection("gate_proj", ".experts.w13_weight", "w1", split=(0, 2)),  # first half
            MoeProjection("up_proj",   ".experts.w13_weight", "w3", split=(1, 2)),  # second half
        ),
    ),
    MoeExpertSpec(
        source_suffix=".mlp.experts.down_proj",
        split_dim=1,
        projections=(
            MoeProjection("down_proj", ".experts.w2_weight", "w2"),  # whole tensor
        ),
    ),
)

FP8_MODEL_SPECS: list[ModelFp8Spec] = [QWEN35_FP8_SPEC, ...]

def resolve_fp8_spec(hf_config) -> ModelFp8Spec | None:
    return next((s for s in FP8_MODEL_SPECS if s.matches(hf_config)), None)

can you help propose some more modular per model abstraction along these lines? Also seeing if any of mappings from megatron-bridge like here: https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/b3a5514340af701340e34b9ec506d92688d2a1b9/src/megatron/bridge/models/qwen/qwen35_bridge.py#L187 can be reused would go a long way in making this more extensible/maintainable in the future

similarly here it would be great to remove any mention of specific models and use some abstraction for layers to ignore - we shouldn't hard code things for qwen3.5 in here